Machine Learning is becoming increasingly more important in today's world. It is therefore very important to provide understanding of the decision-making process of machine-learning models. A popular way to do this is by looking at the Shapley-Values of these models as introduced by Lundberg and Lee. In this thesis, we discuss the two main results by Lundberg and Lee from a mathematically rigorous standpoint and provide full proofs, which are not available from the original material. The first result of this thesis is an axiomatic characterization of the Shapley values in machine learning based on axioms by Young. We show that the Shapley values are the unique explanation to satisfy local accuracy, missingness, symmetry and consistency. Lundberg and Lee claim that the symmetry axiom is redundant for explanations. However, we provide a counterexample that shows the symmetry axiom is in fact essential. The second result shows that we can write the Shapley values as the unique solution to a weighted linear regression problem. This result is proven with the use of dimensionality reduction.

Point-of-Care Ultrasound (POCUS) is the practice of clinicians conducting and interpreting ultrasound scans right at the patient's bedside. However, the expertise needed to interpret these images is considerable and may not always be present in emergency situations. This reality makes algorithms such as machine learning classifiers extremely valuable to augment human decisions. POCUS devices are becoming available at a reasonable cost in the size of a mobile phone. The challenge of turning POCUS devices into life-saving tools is that interpretation of ultrasound images requires specialist training and experience. Unfortunately, the difficulty to obtain positive training images represents an important obstacle to building efficient and accurate classifiers. Hence, the problem we try to investigate is how to explore strategies to increase accuracy of classifiers trained with scarce data. We hypothesize that training with a few data instances may not suffice for classifiers to generalize causing them to overfit. Our approach uses an Explainable AI-Augmented approach to help the algorithm learn more from less and potentially help the classifier better generalize.

Deep Learning has achieved tremendous results by pushing the frontier of automation in diverse domains. Unfortunately, current neural network architectures are not explainable by design. In this paper, we propose a novel method that trains deep hypernetworks to generate explainable linear models. Our models retain the accuracy of black-box deep networks while offering free lunch explainability by design. Specifically, our explainable approach requires the same runtime and memory resources as black-box deep models, ensuring practical feasibility. Through extensive experiments, we demonstrate that our explainable deep networks are as accurate as state-of-the-art classifiers on tabular data. On the other hand, we showcase the interpretability of our method on a recent benchmark by empirically comparing prediction explainers. The experimental results reveal that our models are not only as accurate as their black-box deep-learning counterparts but also as interpretable as state-of-the-art explanation techniques.

Algorithmic approaches to "explaining" model predictions have proliferated as machine learning methods gain in popularity. We refer interested readers to Molnar (2022) or Arrieta et al. (2020) for in-depth surveys. For the purposes of this paper, we simply note that there are many high-level approaches to explaining model predictions and we focus solely on SHAP (Lundberg and Lee (2017)). SHAP is a popular "local feature attribution" method, which means it attempts to explain a model by "scoring" input feature contributions for a specific prediction. Other common examples of local attribution methods include LIME (Ribeiro et al. (2016)), Integrated Gradients (Sundararajan et al. (2017)), and GradCAM (Selvaraju et al. (2017)). While each of these methods deserve detailed study, this paper is a thorough investigation of the statistical properties of SHAP. SHAP applies the Shapley value from game theory (Shapley (1953)) to model explanation by considering features as "players" in a cooperative game. Lundberg and Lee (2017) approximate Shapley values for each feature using a weighted least squares regression, where the regression weights are a transformation of the original Shapley value weights.

Explainable artificial intelligence (XAI) and interpretable machine learning (IML) have become active research fields in recent years (Adadi and Berrada 2018; Molnar 2019). This is a natural consequence as complex machine learning (ML) models are now applied to solve supervised learning problems in many high-risk areas: cancer prognosis (Kourou et al. 2015), credit scoring (Kvamme et al. 2018), and money laundering detection (Jullum, Løland, et al. 2020). The high prediction accuracy of complex ML models often comes at the expense of model interpretability. As the goal of science is to gain knowledge from the collected data, the use of black-box models hinders the understanding of the underlying relationship between the features and the response, and thereby curtail scientific discovery. Model explanation frameworks from the XAI field extract the hidden knowledge about the underlying data structure captured by a black-box model, and thereby make the model's decision-making process transparent. This is crucial for, e.g., medical researchers that apply an ML model to obtain well-performing predictions, but who simultaneously also strive to discover important risk factors. Another driving factor is the Right to Explanation legislation in EU's General Data Protection Regulation (GDPR) (European Commission 2016).

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We establish a new class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 25 existing methods, including several of the most widely used approaches (SHAP, LIME, Meaningful Perturbations, permutation tests). This new class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.

With increasing reliance on the outcomes of black-box models in critical applications, post-hoc explainability tools that do not require access to the model internals are often used to enable humans understand and trust these models. In particular, we focus on the class of methods that can reveal the influence of input features on the predicted outputs. Despite their wide-spread adoption, existing methods are known to suffer from one or more of the following challenges: computational complexities, large uncertainties and most importantly, inability to handle real-world domain shifts. In this paper, we propose PRoFILE, a novel feature importance estimation method that addresses all these challenges. Through the use of a loss estimator jointly trained with the predictive model and a causal objective, PRoFILE can accurately estimate the feature importance scores even under complex distribution shifts, without any additional re-training. To this end, we also develop learning strategies for training the loss estimator, namely contrastive and dropout calibration, and find that it can effectively detect distribution shifts. Using empirical studies on several benchmark image and non-image data, we show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.

A feature-based model explanation denotes how much each input feature contributes to a model's output for a given data point. As the number of proposed explanation functions grows, we lack quantitative evaluation criteria to help practitioners know when to use which explanation function. This paper proposes quantitative evaluation criteria for feature-based explanations: low sensitivity, high faithfulness, and low complexity. We devise a framework for aggregating explanation functions. We develop a procedure for learning an aggregate explanation function with lower complexity and then derive a new aggregate Shapley value explanation function that minimizes sensitivity.

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP . Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to'improve' SHAP in a way that we believe to be flawed. Further, our criticism concerns TreeExplainer in SHAP, which really uses conditional expectations (without approximating them by unconditional ones).